Point Cloud Hand–Object Segmentation Using Multimodal Imaging with Thermal and Color Data for Safe Robotic Object Handover

This paper presents an application of neural networks operating on multimodal 3D data (3D point cloud, RGB, thermal) to effectively and precisely segment human hands objects held in hand realize a safe human–robot object handover. We discuss the problems encountered building sensor system, while focus is calibration alignment set cameras including thermal, NIR cameras. propose use copper–plastic chessboard target with internal active light source (near-infrared visible light). By brief heating, could be simultaneously legibly captured by all Based dataset our PointNet, PointNet++, RandLA-Net are utilized verify effectiveness applying cloud for hand–object segmentation. These were trained various modes (XYZ, XYZ-T, XYZ-RGB, XYZ-RGB-T). The experimental results show significant improvement segmentation performance XYZ-RGB-T (mean Intersection over Union: 82.8% RandLA-Net) compared other three (77.3% 35.7% XYZ), which it worth mentioning that Union single class achieves 92.6%.